Streaming application visuals using page-like splitting of individual windows

ABSTRACT

The disclosure relates to the transfer of visuals (e.g., window visuals) over virtual frames that may be stored in any number of video frames of one or more video streams. The visuals may be split into two-dimensional (2D) pages of a virtual frame, with each of the 2D pages being a fraction of the size of video frames of the video stream(s). The virtual frame may be encoded to the video frames of the video stream(s) and later reconstructed in accordance with a page table.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Nonprovisional applicationSer. No. 16/441,578, filed on Jun. 14, 2019, which claims the benefit ofU.S. Provisional Application No. 62/751,366, filed on Oct. 26, 2018,each of which is hereby incorporated by reference in its entirety.

BACKGROUND

Conventional approaches for remote computing may provide a user withaccess to software running on a remote computing device (e.g., a server)by streaming an entire desktop of the remote computing device to a localdevice of the user (e.g., a client). As a result, the user of the localdevice may only have the option of viewing a representation of theentire remote desktop on the local device, either within a single windowor as a full-screen view. To access the software of the remote computingdevice, the user must interact with the entire remote desktop beingstreamed on the local device of the user. However, a user may beinterested in viewing and/or interacting with select windows or only asubset of the entire desktop of the remote computing device. In suchscenarios, these conventional approaches that produce the entire remotedesktop unnecessarily consume excess bandwidth, processing power, andenergy to generate and transmit the remote desktop to the local device.Moreover, security concerns may arise in cases where a user should notbe granted full access to an entire remote desktop, but only to acertain set of remote applications.

Additionally, conventional systems may not offer seamless integrationbetween the remote desktop and the local desktop of the local device.For example, where the remote desktop is streamed into a window, thewindow that displays the remote desktop may allow some level ofinteraction between the remote and local desktops (e.g., drag and dropfunctionality), but may always separate application windows from withinthe remote desktop from application windows of the local desktop. Asanother example, when interacting with any application of the remotedesktop, the entire remote desktop window may be displayed above (over)or below (under) each of the application windows in the Z-order of thelocal desktop. Further, while some systems may stream a client area of asingle application window, the non-client area (the graphical contentbeyond what is directly contained in the client area, such as a windowframe, and any functionality provided through user interface widgets onthe window frame) may not be made available to the client desktop.

SUMMARY

Aspects of the disclosure provide for the transfer of visuals (e.g.,window visuals) over virtual frames that may be stored in any number ofvideo frames of one or more video streams. The visuals may be split intotwo-dimensional (2D) pages of a virtual frame, with each of the 2D pagesbeing a fraction of the size of video frames of the video stream(s). Thevirtual frame may be encoded to the video frames of the video stream(s)and later reconstructed in accordance with a page table.

BRIEF DESCRIPTION OF THE DRAWINGS

The present systems and methods for individual application windowstreaming suitable for remote desktop applications is described indetail below with reference to the attached drawing figures, wherein:

FIG. 1 is a diagram of an example desktop streaming system, inaccordance with some embodiments of the present disclosure;

FIG. 2A is an example of a remote desktop, in accordance with someembodiments of the present disclosure;

FIG. 2B is an example of a local desktop, in accordance with someembodiments of the present disclosure

FIG. 3 is an illustration used to describe examples of generating visualsurfaces for inclusion in a network stream, in accordance with someembodiments of the present disclosure;

FIG. 4A is an illustration of converting alpha information to a YUV420video frame format in an RGB color space, in accordance with someembodiments of the present disclosure;

FIG. 4B is an illustration of an example of laying out image regions ofan image with per-pixel alpha information and of a display of the imageon a desktop, in accordance with some embodiments of the presentdisclosure;

FIG. 5 is an illustration of generating a frame of a virtual frame fromwindow visuals of application windows, in accordance with someembodiments of the present disclosure;

FIG. 6 is a flow diagram showing a method for remote desktop streaming,in accordance with some embodiments of the present disclosure;

FIG. 7 is a flow diagram showing a method rendering desktop visuals of aremote desktop on a local desktop, in accordance with some embodimentsof the present disclosure;

FIG. 8 is a flow diagram showing a method for selectively transmittingportions of a remote desktop to a local desktop, in accordance with someembodiments of the present disclosure;

FIG. 9 is a flow diagram showing a method for transmitting alphainformation of visuals in one or more frames of a video stream, inaccordance with some embodiments of the present disclosure;

FIG. 10 is a flow diagram showing a method for streaming a virtual frameover a video stream, in accordance with some embodiments of the presentdisclosure;

FIG. 11 is an example operating environment of in accordance with someembodiments of the present disclosure; and

FIG. 12 is a block diagram of an example computing device suitable foruse in implementing some embodiments of the present disclosure.

DETAILED DESCRIPTION

Systems and methods are disclosed related to individual applicationwindow streaming suitable for remote desktop applications. Morespecifically, systems and methods are disclosure related to—in lieu ofstreaming an entire remote desktop to a local desktop—streaming one ormore application windows and/or other desktop elements of the remotedesktop to the local desktop.

In contrast to conventional approaches, disclosed approaches allow forany number of application windows or other desktop elements of a remotedesktop to be streamed to the desktop of a local device withoutincluding the entire remote desktop in the stream. To do this, windowvisuals and corresponding window properties of each relevant window ofthe remote desktop may be captured and encoded to a video stream of thestream (e.g., using a standard such as Real-Time Transport Protocol(RTP) or a proprietary protocol type). Data used to represent theirrelevant and/or unselected windows or other desktop areas may beminimized, such as by replacing the data with lower entropy data (e.g.,black pixel data) that may be highly compressed in the video streamand/or replacing those areas with other visual content. Further, thevideo stream comprising the window visuals and window metadatarepresentative of the window properties may be transmitted to the localdevice. By streaming only a set of particular application windows and/orother desktop elements to a local device, as well as leveraging videostreaming technologies, the disclosed approaches require less bandwidth,processing, and power compared to conventional systems. In addition, incontrast to security concerns present in conventional systems, such asthose described herein, the approaches of the present disclosure offersecurity improvements, by excluding from the stream application windows,desktop elements, or visuals that are sensitive in nature, for example.

Window visuals and window properties of one or more application windowsmay be received and decoded from the stream to create local windowvisuals on the local desktop. Using the received window properties, thedecoded window visuals may be rendered and displayed on the localdesktop in a proxy window for each of the corresponding remote windows.In one or more embodiments, a proxy window may include decoded windowvisuals that extend beyond what is in a client area. The decoded windowvisuals and window properties of any number of application windows maybe received in a single video stream or multiple video streams, over asingle network stream, while allowing for interaction with theindividual proxy windows, as if they were native to the local desktop.Thus, in contrast to conventional systems, by rendering a proxy windowon the local desktop from the window visuals and window properties,disclosed approaches allow the application windows to be seamlesslyintegrated into the local desktop.

In some examples, a user or the system may determine at least oneapplication window of a remote desktop of interest to be streamed to alocal desktop of a local device. The portion of the remote desktop notidentified or otherwise determined to be of interest may be excludedfrom or minimized in the stream. For example, the unselected portion ofthe remote desktop may be filtered out or converted to a minimalrepresentation using lower entropy data from a first (e.g., captured)visual surface in order to generate a second (e.g., to be transmitted)visual surface that still includes fully represented window visuals ofthe portions of the remote desktop that are determined to be ofinterest. The second visual surface may then be encoded to the streamand transmitted to the local device. In some examples, the portion(s) ofthe remote desktop may be removed by applying a stencil to the first(e.g., captured) visual surface to remove, or overwrite, at least someof the portion(s) of the remote desktop while retaining window visualsof relevant windows and other desktop elements. Removing and/ormodifying a portion of the remote desktop so that it is not transmittedin the stream or transmitted using less bandwidth and processing powermay also provide increased security in addition to bandwidth andprocessing savings by both the remote computing device encoding thestream and the local device decoding the stream.

Further aspects of the disclosure may provide for the transfer ofper-pixel transparency information (alternatively referred to as “alphainformation”) using video codecs (e.g., H.264) that do not define analpha channel. For example, during an encoding phase at the remotecomputing device, alpha information of window visuals of an applicationwindow may be transcoded into the supported channels of a video streamto generate samples of supported color space that are representative ofthe alpha information. Additionally, triplets (blocks of 1×3 or 2×3) ofthe alpha samples may be attached to the video stream as a single samplein the color space. In some non-limiting examples, the color space maybe a Luma-Chrominance (YUV) color space or a Red Green Blue (RGB) colorspace. By transcoding alpha information to a supported color space,transparency information for application windows may be transported in avideo stream that otherwise may not support the transparencyinformation.

Additional disclosed approaches may provide for a header of a videostream (e.g., using an RTP header extension for a stream transportedusing RTP) to be used to transmit window properties of at least oneapplication window to a local device. For example, the window propertiesmay be represented by window metadata that is included in the header. Byencoding the window properties in the header, the window properties andwindow visuals may be transported in the same video stream—therebyresulting in bandwidth savings and reduced system complexity, withoutrequiring additional dedicated streams.

In further examples, window metadata representative of the windowproperties may be delta-encoded in a stream for transmission to thelocal device. The window metadata may be transmitted in the stream usingstructured byte-wise diffs, where structured metadata is processedmember-by-member, byte-by-byte, to delta-encode the window properties ofan application window. Sending delta information (e.g., changes in thewindow properties) rather than all the current window propertyinformation in the stream may allow for significantly reduced bandwidthutilization and may reduce processing requirements because less dataneeds to be sent and processed to convey the same information.

To encode window visuals of application windows into a video stream,disclosed approaches may include splitting the window visuals of atleast one application window into 2D pages of a virtual frame—with eachof the 2D pages being a fraction of a size of frames of a video stream.The virtual frame may be encoded to the frames of the stream. In someexamples, the virtual frames may be used in conjunction with an appliedstencil. For example, the stencil may be used to generate a second(e.g., to be transmitted) visual surface that may then be split into the2D pages of the virtual frame. In further examples, the stencil may ormay not be used, and a set of application windows and/or other desktopelements may be captured directly (e.g., from a remote operatingsystem), and then split into the 2D pages of the virtual frame.Streaming “paged” window visuals may simplify the logic applied to theplacement of multiple windows into a single transmitted surface andtheir later reconstruction from that surface and may also avoid theusage of complex bin-packing algorithms. In some examples, this approachmay be used to transmit one or more portions of the 2D pages rather thancomplete window visuals in every frame, which may allow for balancingworkload and bandwidth requirements. Further, this approach may be usedto mitigate video frame size limitations of video encoders used toencode the video stream.

Now referring to FIG. 1, FIG. 1 is a diagram of an example desktopstreaming system 100, in accordance with some embodiments of the presentdisclosure. It should be understood that this and other arrangementsdescribed herein are set forth only as examples. Other arrangements andelements (e.g., machines, interfaces, functions orders, and groupings offunctions etc.) may be used in addition to or instead of those shown,and some elements may be omitted altogether for the sake of clarity.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distribute components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed by oneor more entities may be carried out by hardware, firmware, and/orsoftware. For instance, some functions may be carried out by a processorexecuting instructions stored in memory.

The desktop streaming system 100 may include, for example, a remotesystem 116 and a local system 118. The remote system 116 may include,for example, a communications manager 120A, a stream encoder 124, astreaming determiner 126, a virtual frame generator 128, a desktopcapturer 130, and a desktop event monitor 132A. The local system 118 mayinclude, for example, a communications manager 120B, a desktop renderer136, a desktop event monitor 132B, and a stream decoder 140.

As an overview, the desktop streaming system 100 may be used to streamone or more portions of a remote desktop 102A to a local desktop 102B.For example, the remote system 116 may capture desktop visuals of theremote desktop 102A and associated desktop properties for inclusion in anetwork stream 134. Desktop visuals and desktop properties may beincluded in desktop data that may be associated with a particularportion or element of a desktop. Desktop visuals of the portion of thedesktop may refer to visual content, and a desktop property may refer todata that is descriptive of the portion of the desktop (e.g.,dimensions, location, etc.). In various examples, the portion of thedesktop may be an application window—such as an application window106A—in which case the desktop data may be referred to as window data,the desktop visuals may be referred to as window visuals, and thedesktop properties may be referred to as window properties. In someembodiments, to capture the desktop visuals, the remote system 116captures the remote desktop 102A and applies a stencil to the capturedremote desktop 102A to remove at least some of a desktop background 108A(e.g., by overwriting corresponding desktop visuals with lower entropydata) and/or other extraneous information while retaining desktopvisuals of one or more application windows—such as the applicationwindow 106A. The local system 118 may use the desktop visuals anddesktop properties from the network stream 134 to render the portion(s)of the remote desktop 102A on the local desktop 102B. For example, forthe application window 106A, the local system 118 may render a proxywindow 106B on the local desktop 102B using corresponding window visualsand window properties from the network stream 134. The proxy window 106Bmay be integrated into the local desktop 102B with one or more nativeapplication windows—such as a native application window 114. Unselectedportions of the desktop, such as application window 104, will be leftout from the transmitted stream.

The communications manager 120A of the remote system 116 may beconfigured to manage communications provided by the remote system 116,such as in the network stream 134 of the desktop streaming system 100(e.g., comprising data representative of window visuals and windowproperties of at least one application window). The communicationsmanager 120A of the remote system 116 may also be configured to managecommunications provided to the remote system 116 (e.g., comprising datarepresentative of window events for application window(s) of the remotedesktop 102A, which may also be in the network stream 134). Thestreaming determiner 126 may be configured to determine at least oneapplication window and/or other portions of the remote desktop 102A toinclude in the network stream 134. The desktop capturer 130 may beconfigured to capture the portion(s) of the remote desktop 102A, such aswindow visuals and corresponding window properties of the at least oneapplication window (e.g., the application window 106A). The streamencoder 124 may be configured to encode data representative of theportions of the remote desktop, such as the window visuals andcorresponding window properties of the at least one application windowto the network stream 134 (e.g., a video stream of the network stream134). In some embodiments, the virtual frame generator 128 may be usedto split the window visuals of the at least one application window intotwo-dimensional (2D) pages of a virtual frame for inclusion in thenetwork stream 134. The desktop event monitor 132A may be configured toidentify desktop events, such as a window creation event or a windowdestruction event associated with the remote desktop 102A (e.g.,indicated by user input to the local desktop 102B). The desktop eventmonitor 132A may be further configured to update a list of applicationwindows to include in the network stream 134 based, at least in part, onthe identified desktop events.

The communications manager 120B of the local system 118 may beconfigured to manage communications provided by the local system118—such as in the network stream 134 of the desktop streaming system100—and/or provided to the local system 118. The stream decoder 140 maybe configured to decode data from the network stream 134, such as datarepresentative of desktop visuals and desktop properties. The desktoprenderer 136 may be configured to render, on the local desktop 102B, theportion(s) of the remote desktop 102A included in the network stream 134using the decoded data. For example, for each application window, thedesktop renderer 136 may render (e.g., generate) a corresponding proxywindow using the window properties of the application window so thatthey may appear on the local desktop 102B as they do on the remotedesktop 102A. The desktop event monitor 132B may be configured toidentify events associated with the remote desktop 102A, such as thewindow creation event or the window destruction event (e.g., when userinput selects a control element to close a proxy window).

The components of FIG. 1 may generally be implemented using anycombination of a user device(s) 1106 and/or a server device(s) 1102 ofFIG. 11. In some examples, the remote system 116 may be implemented onone or more of the server devices 1102 and the local system 118 may beimplemented on the user device 1106. In other examples, both the localsystem 118 and the remote system 116 may be implemented on separate userdevices 1106. The remote system 116 and/or the local system 118 may eachbe provided via multiple devices arranged in a distributed environmentthat collectively provide the functionality described herein or may eachbe embodied on a single device. Thus, while some examples used todescribe the desktop streaming system 100 may refer to particulardevices and/or configurations, it is contemplated that those examplesmay be more generally applicable to the potential combinations ofdevices and configurations described above.

As mentioned herein, the communications manager 120A and/or thecommunications manager 120B may be configured to manage communications(e.g., of the network stream 134) received by the remote system 116 orthe local system 118 of the desktop streaming system 100 and/or providedby the remote system 116 or the local system 118 of the desktopstreaming system 100. The communications manager 120A and/or thecommunications manager 120B may also be configured to managecommunications within their respective systems. In such examples, one ormore of the communications may be transmitted between components of acomputing device 1200 over a bus 1202 of FIG. 12.

Where a communication is received and/or provided as a networkcommunication, the communications manager 120A and/or 120B may include anetwork interface that may use one or more wireless antenna(s) and/ormodem(s) to communicate over one or more networks.

The streaming determiner 126 of the remote system 116 may determine atleast one application window or other portion of the remote desktop 102Ato include in the network stream 134 to the local desktop 102B. Thisdetermination may be based on user input and/or other user-definedsettings and/or system-defined information, such as a configurationfile. For example, a user may use a graphical user interface (GUI) viathe remote system 116 and/or the local system 118 to provide input thatselects which application(s) and/or application windows associated withthe application(s) to stream from the remote desktop 102A to the localdesktop 102B. For example, a user may select an application to indicateto the streaming determiner 126 to automatically include eachapplication window associated with that particular application in thenetwork stream 134. Additionally or alternatively, user input mayindividually select specific application windows for the streamingdeterminer 126 to include in the network stream 134. In some examples,only those application windows selected by the user may be streamed tothe local desktop 102B and/or be displayed on the local desktop 102B.

Additionally, at least some of the settings regarding which applicationwindows and/or other portions of the remote desktop 102A to include inthe network stream 134 may be system-defined. For example, the streamingdeterminer 126 may determine which application windows and/or otherdesktop elements are associated with the remote desktop 102A. To do so,the streaming determiner 126 may query (e.g., using one or more systemAPI calls) a service of an operating system or metadata maintained bythe operating system that hosts the remote desktop 102A (e.g., a windowmanager). The streaming determiner 126 may or may not be part of thehost operating system depending on the embodiment. The streamingdeterminer 126 may use the results of the query to determine one or moreapplication windows to stream. For example, the results of the query mayindicate whether an application window is visible on the remote desktop102A, and the streaming determiner 126 may stream the application windowbased at least in part on the application window being visible. As anexample, the streaming determiner 126 may decide to only include visibleapplication windows (and/or window visuals thereof) in the networkstream 134 to the local desktop 102B.

The desktop capturer 130 of the remote system 116 may be configured tocapture desktop visuals of the remote desktop 102A and correspondingdesktop properties. For example, the desktop capturer 130 may capturethe desktop visuals and corresponding desktop properties for eachportion of the remote desktop 102A that the streaming determiner 126determines to include in the network stream 134.

FIG. 2A is an illustration including an example of the remote desktop102A, in accordance with some embodiments of the present disclosure.Desktop visuals of the remote desktop 102A which may be captured by thedesktop capturer 130 may include visuals of windows (e.g., of theapplication window 106A and the application window 216A), icons (e.g.,an icon 202A), taskbars (e.g., a taskbar 210A), command buttons (e.g., acommand button 212), system trays, start buttons, start menus,shortcuts, cursors (e.g., a cursor 208A), window previews (e.g., apreview window 204A), desktop backgrounds (e.g., the desktop background108A), and/or other elements that may be present on the remote desktop102A. Window visuals for an application window may include text orgraphics displayed as outputs in a client area of an application window,such as a client area 206B of the application window 106A. Additionally,window visuals for an application window may include text or graphics ofone or more system areas or frame areas of an application window, suchas a system area 206A of the application window 106A. Examples of asystem area or a frame area include at least one of: a title bar, a menubar, a window menu, a minimize button, a maximize button, a sizingborder, and/or a scrolling bar.

Corresponding desktop properties for visuals of windows, icons,taskbars, command buttons, system trays, start buttons, start menus,shortcuts, desktop backgrounds, and/or other visual elements may includeone identifier and one or more of: a caption, geometry, style, location(e.g., coordinates), window mode (e.g., full screen, windowed, etc.),and/or other property associated with the portion of the remote desktop.In some examples, a desktop property may indicate a type of the desktopvisuals, such as whether the desktop visuals correspond to anapplication window, window preview, etc.

The desktop capturer 130 may capture the corresponding desktopproperties for desktop visuals from a remote desktop operating system,service, desktop manager software, file, memory, window manager,application, and/or other suitable source. For example, one or morewindow properties of an application window(s) captured for the networkstream 134 may correspond to one or more window properties used tocompose the application window(s) on the remote desktop 102A. As furtherexamples, one or more of the desktop properties may be determined basedat least in part on computer vision and/or object tracking algorithms(e.g., performed on a screen capture of the remote desktop 102A).

In some examples, the desktop capturer 130 may capture at least some ofthe desktop visuals into one or more visual surfaces and encode the oneor more visual surfaces to the network stream 134. As an example, one ofthe visual surfaces may be a remote desktop surface that isrepresentative of the remote desktop 102A, such as a screen shot orscreen capture of at least a portion of the remote desktop 102A and/or adisplay of a remote device that displays the remote desktop 102A. Asfurther examples, one or more of the visual surfaces may be a desktopcomposition surface that is representative of desktop visuals used bythe operating system and/or window manager to compose the remote desktop102A. As an example, a desktop composition surface may include at leasta portion of an individual application window and/or other visualelement. For example, a desktop composition surface for the applicationwindow 106A may include the client area 206B and the system area 206A.

The desktop capturer 130 may use the streaming determiner 126 todetermine at least a portion of the remote desktop 102A to exclude fromthe network stream 134. For example, the desktop capturer 130 may usethe streaming determiner 126 to determine at least one applicationwindow and/or another portion of the remote desktop 102A to exclude fromthe network stream 134. In the example of FIG. 2A, this may include theremote desktop background 108A and an application window 214. Further,the desktop capturer 130 may determine at least some of the portion(s)of the remote desktop 102A to remove from the visual surface (e.g., arepresentation of the remote desktop 102A). In various embodiments, theportions of the remote desktop 102A that the desktop capturer 130removes do not include the portion(s) of the remote desktop 102A thatthe streaming determiner 126 determines or selects to include in thenetwork stream 134. As a result, the other visual surface(s) may includedesktop visuals of portions of the remote desktop 102A that thestreaming determiner 126 determines to include in the network stream 134without at least some of the other portions of the remote desktop 102A.The stream encoder 124 may encode the other visual surface(s) in thenetwork stream 134.

In some examples, the desktop capturer 130 removes at least some of theportion(s) of the remote desktop 102A from the visual surface(s) usingat least one stencil. For example, the desktop capturer 130 may apply astencil to a visual surface (e.g., a remote desktop surface) to remove(e.g., overwrite) at least some visual content resulting in a visualsurface that does not include the removed visual content. A stencil mayidentify—on a visual surface—which portions of the remote desktop 102Aare to be included in the network stream 134 and which portions of theremote desktop are to be excluded from the network stream 134. Thestencil may be used to remove and replace at least some image data(e.g., pixel data) of non-session areas (those areas that do notcorrespond to a portion of the remote desktop 102A to be included in thenetwork stream 134) with other image data. The desktop capturer 130 mayconstruct the stencil, for example, by traversing all visible windows ofall streamed processes as specified by the streaming determiner 126.

In some examples, at least some of the image data representative of animage of the remote desktop 102A may be replaced with at least someimage data representative of another image. For example, image datarepresentative of the desktop background 108A (FIG. 1) may be replacedwith image data representative of a desktop background 108B.Additionally, or alternatively, at least some of the image data ofnon-session areas may be replaced with lower entropy image data—such aszero or near-zero entropy data. Examples of zero or near-zero entropydata may include data representative of a common pixel color, such asblack. Generally, any area having a solid fill color may comprise lowentropy data. For example, the desktop background 108A may be replacedwith image data representative of black pixels, and the desktopbackground 108B, shown in FIG. 1, may be a native desktop background ofthe local desktop 102B. Reducing the entropy of image data in one ormore regions of a visual surface(s) that are not of interest for displayon the local desktop 102B may allow the stream encoder 124 to compressthe visual surface(s) to a smaller size, reducing the bandwidth requiredto stream the remote desktop 102A to the local desktop 102B. Forexample, the desktop renderer 136 may not render image data from thenetwork stream 134 that corresponds to the non-session areas of theremote desktop 102A on the local desktop 102B. The desktop renderer 136may instead render other image data or may not render any image data inthose non-session areas; in such cases, the operating system of thelocal desktop may render non-session areas when the desktop renderer 136is not part of the operating system.

FIG. 3 is used to describe examples of the desktop capturer 130generating visual surfaces for inclusion in the network stream 134 usingstenciling and/or other approaches. Referring now to FIG. 3, the top rowis an illustration including examples of remote desktop surfaces 300, inaccordance with some embodiments of the present disclosure; the bottomrow is an illustration including examples of visual surfaces 340 whichmay correspond to the remote desktop surfaces 300 of FIG. 3, inaccordance with some embodiments of the present disclosure. A remotedesktop surface 302, a remote desktop surface 304, a remote desktopsurface 306, and a remote desktop surface 308 may correspond to a first,second, third and fourth remote desktop, respectively. A visual surface332, a visual surface 334, a visual surface 336, and a visual surface338 may be generated by the desktop capturer 130 from the remote desktopsurface 302, the remote desktop surface 304, the remote desktop surface306, and the remote desktop surface 308, respectively, for inclusion inthe network stream 134. The visual surface 332, the visual surface 334,the visual surface 336, and the visual surface 338 may representexamples of full-screen surfaces of the remote desktop 102A and/or mayrepresent examples of full-screen surfaces for display on the localdesktop 102B.

The remote desktop surface 302 includes window visuals for applicationwindows 312, 314, and 316. As an example, the application windows 312,314, and 316 may correspond to the same application, with theapplication window 312 comprising a primary work area and theapplication windows 314 and 316 comprising toolbars used to select toolsfor interacting with the primary work area. In some examples, thestreaming determiner 126 selects each of the application windows 312,314, and 316 for inclusion in the network stream 134. As the applicationwindows 312, 314, and 316 fill the entire remote desktop 102A, thevisual surface 332 may be the same as or substantially similar to theremote desktop surface 302 (e.g., the same application windows in theircorresponding locations, but optionally without one or more systemdesktop elements, such as a mouse cursor, taskbar, etc.). The remotedesktop surface 304 includes a desktop region 310 and applicationwindows 318 and 320. The application window 320 may correspond to awindow of an application, such as an inbox of an email application, andthe application window 318 may correspond to a different application,such as a media player, or it may be another application window of theapplication that corresponds to the application window 320. The desktopregion 310 may include desktop visuals of at least some of the desktopbackground 108A and optionally of other elements that are forpresentation over the desktop background 108A, such as desktop icons,text, a system menu, and/or a system taskbar.

In some examples, the streaming determiner 126 selects each of theapplication windows 318 and 320 for inclusion in the network stream 134.The desktop capturer 130 may, as a result, apply a stencil to the remotedesktop surface 304 to generate the visual surface 334, where image datacorresponding to the desktop region 310 is replaced with lower entropyimage data (e.g., representative of black pixels) and/or other imagedata (e.g., representative of a different image) representative of aregion 330. The remote desktop surface 306 includes an applicationwindow 322 in a full-screen mode. Thus, the visual surface 336 may bethe same as or substantially similar to the remote desktop surface 306.The remote desktop surface 308 includes only the desktop region 310.Thus, the desktop capturer 130 may apply a stencil to the remote desktopsurface 308 to generate the visual surface 338, where image datacorresponding to the desktop region 310 is replaced with lower entropyimage data (e.g., representative of black pixels) and/or other imagedata (e.g., representative of a different image) representative of theregion 330. In some examples, the stencil may not be used and thedesktop capturer 130 may generate the visual surface 338 without usingand/or capturing the remote desktop surface 308. This may occur, forexample, when the streaming determiner 126 determines that the remotedesktop 102A does not include any desktop visuals for display on thelocal desktop 102B.

In some examples, the remote desktop surfaces 302, 304, 306, and 308correspond to multiple displays of the same remote system 116. Forexample, each remote desktop surface may be rendered for a respectiveportion of the remote desktop visible on a respective monitor presentedby the remote system 116 using a remote user device. Desktop visuals anddesktop properties for each portion of the remote desktop may beincluded in the network stream 134 (e.g., in the same video stream) ormay have separate streams (e.g., separate video streams per display withper-stream window visuals, window properties, and timing information toenable client-side window composition from asynchronous video streams).Further, the remote desktop surfaces 302, 304, 306, and 308 may be oneremote desktop surface and/or the visual surfaces 332, 334, 336, and 338may be one visual surface, by way of example. In addition, while thevisual surfaces 332, 334, 336, and 338 are described as being generatedusing stencils applied to the remote desktop surfaces 302, 304, 306, and308, any suitable approach may be used for various embodiments. Forexample, any number of remote desktop surfaces and/or stencils may beused to generate a visual surface for inclusion in the network stream134.

Additionally, any number of intermediate visual surfaces may be used togenerate the visual surface. In some examples, any of the visualsurfaces 332, 334, 336, and 338 may be generated by the desktop capturer130 without use of a remote desktop surface or portion thereof. Forexample, the desktop capturer 130 may generate the visual surfaces 332,334, 336, and 338 by compositing image data for different portions ofthe remote desktop 102A that are to be included in the network stream134 (e.g., where the desktop capturer 130 has accesses to individualsurfaces the operating system uses to composite the remote desktop 102Afor display).

The stream encoder 124 may be configured to encode data representativeof the portions of the remote desktop, such as the desktop visuals andcorresponding desktop properties of the at least one application windowto the network stream 134. To do so, the stream encoder 124 encodes thecaptured desktop visuals to the network stream 134. For example, thestream encoder 124 may encode the any of the various visual surfaces332, 334, 336, and 338 into network stream 134. In one or moreembodiments, the visual surface 332 may be encoded into a single videoframe. However, in some cases, the visual surface 332 may be encodedinto a virtual frame using the virtual frame generator 128, which mayuse the visual surface 332 to generate multiple video frames asdescribed with respect to FIG. 5. As another example, the desktopcapturer 130 may use the virtual frame generator 128 to capture thedesktop visuals that correspond to the visual surface 332 into videoframes of a virtual frame without first capturing the image data to asingle visual surface.

The stream encoder 124 may be configured to encode the desktopproperties corresponding to the desktop visuals into the network stream134 as desktop metadata representative of the desktop properties for thedesktop visuals. The stream encoder 124 may encode the desktopproperties on a per-frame basis in association with correspondingdesktop visuals. For example, the stream encoder 124 may associate thedesktop visuals with corresponding desktop properties using temporaland/or spatial information, such as with associated timestamps,coordinates, frames, and/or frame identifiers. Each video frame may beannotated by a timestamp, which may be used by the desktop renderer 136of the local system 118 to update the desktop state and desktop visualsfrom asynchronous video streams of the network stream 134 (e.g., formulti-monitor use cases). In some examples, the stream encoder 124encodes the window properties to header data of network communicationsof the video stream. For example, the desktop metadata may be includedin a header extension of the transport protocol used to encode thenetwork stream 134. Thus, the desktop metadata may be transported alongwith the desktop visual data, without requiring a separate stream orconnection.

The desktop metadata may be transmitted in the network stream 134 usingdelta-encoding of the desktop properties of the desktop elements. Toreduce network bandwidth consumption, only metadata that has beenchanged since the last packet update may be sent. To prevent losses, areference invalidation scheme may be employed. Likewise, to preventprotocol breaks, one or more future-proofing mechanisms (e.g.,serialized, structured byte-wise diffs, etc.) may be implemented. Forexample, the stream encoder 124 may delta-encode the desktop propertiesusing, for example and without limitation, structured byte-wise diffsthat may bring significant bandwidth savings and employ a versioningscheme for forward/backward compatibility.

Embodiments of the present disclosure provide approaches for streamingwindow metadata on a per-frame basis that allowsforward/backward-compatible delta-coding of “generic” data. The streamencoder 124 may implement these approaches in some embodiments. In oneor more embodiments, data may be described as “generic” if it usesgeneric C/C++ data type definitions, e.g., to avoid additionaldependencies and/or redundant data type representations. The solutionsdescribed herein may be forward/backward-compatible such that clientscan understand outdated or disregard future data definitions ofolder/newer servers. Delta-coding may be used to reduce bandwidthrequirements of rather low-entropy window metadata.

Reducing the amount of data transported over a network may be solved bydiff/patch-based delta-coding. Since the efficiency of a simple“serialize, then send byte-wise diff” approach may not scale well forstructured data involving entries/members of variable size, embodimentsmay leverage C++ templates to flatten data structures toentry/member/byte-wise diffs along template function call hierarchies:generic template function definition which calls delegateimplementations for generic/custom types; specialized implementationsfor arithmetic types using byte-wise diff/patch; specialized (partial)implementations for essential containers (e.g., string, vector, map);specialized delegate implementations for generic/custom types whichsimply enumerate members to diff/patch implementations of the abovebasic types.

These approaches may also support polymorphisms and may scale toarbitrarily structured data either by limiting data types to thesupported ones, or by adding specialized diff/patch implementations.Both the described byte-wise diff and struct-diff implementations mayemploy additional optimizations applicable to diff string size andexecution speed, in addition to the features described.

Regarding forward/backward-compatibility, since enhancing structs bystruct size entries may enable skipping of future/unknown entries whenmerging a diff string, entry deprecation is virtually for free: bysimply never changing the value of a default-constructed entry, thisentry will always account as an incremented skip count in a diff string.

Further aspects of the disclosure provide approaches that may be usedfor the transfer of transparency information for visuals, such asdesktop visuals, using image data of a video stream, even where a videocodec used to encode the video stream does not explicitly support analpha channel. The stream encoder 124 may use these approaches to encodethe desktop visuals of the network stream 134. For example, while asystem desktop is typically opaque, individual windows and/or otherdesktop elements may be at least partially transparent. To illustratethe forgoing, the system area 206A and/or the client area 206B of theapplication window 106A may be semi-transparent, such as when theapplication window 106A is not an active window. As further examples,portions of desktop visuals may be completely transparent. Such contentincludes windows of non-rectangular shape that typically use per-pixeltransparency to achieve such shapes. Common use cases include gamelaunchers or splash screens of unusual and/or asymmetrical shape, andthe rounded rectangle shape of restored application windows.

The transfer of such transparency information may require a remoteapplication streaming solution to stream per-pixel transparencyinformation (e.g., using an alpha channel). However, contemporary videocodecs, such as H.264 (AVC), do not support an alpha channel. Disclosedapproaches may provide for alpha information of desktop visuals ofdesktop elements to be transcoded to a color space of a video stream tostore alpha samples that are representative of the alpha information inthe color space. In various examples, the color space may be aLuma-Chrominance (YUV) color space or a Red Green Blue (RGB) colorspace. Disclosed approaches may enable RGBA streaming using, forexample, an alpha-agnostic YUV444/420 video streaming pipeline byattaching appropriately converted alpha information to YUV444/420 videoframes—either in the RGB or YUV space—and having the underlying videocodec process alpha values, or samples, as regular RGB or YUV values.However, any suitable streaming format and color spaces may be employedwithout departing from the scope of the present disclosure.

In various embodiments, each alpha sample may define transparencyinformation for a particular pixel. Alpha samples may be stored in oneor more color channels that define the color space for the video stream,along with color samples that are typically stored in those colorchannels (e.g., RGB channels or YUV channels). In some examples, whenrendering the image data stored in the video stream, an image renderer(e.g., the desktop renderer 136 of the local system 118) may use adecoder (e.g., the stream decoder 140 of the local system 118) that iscapable of distinguishing between the color samples and the alphasamples. Further, the decoder may be capable of determining which alphasamples are associated with which color samples in order to render acorresponding pixel. These associations, for example, may be implicit inthe locations of the alpha samples relative to the color samples.

In some examples, groups of alpha samples may be attached to the videostream as a single sample of the color space (e.g., a single pixel colorvalue). In such examples, for some of the color samples of the colorspace, each value that typically represents a color component of thecolor sample for a pixel may instead represent an alpha component for apixel. Thus, alpha samples for multiple pixels may be stored in thespace typically used to store a color sample for a single pixel. Anysuitable format could be used to store alpha samples in channels ofcolor samples, which may vary for different color spaces. For convertingalpha information to a YUV444 video frame format in an RGB color space,the stream encoder 124 may attach triplets of alpha samples as a singleRGB sample: a0 to r0; a1 to g0; and a2 to b0, as in an image 404 of FIG.4B. Referring now to FIG. 4A, FIG. 4A is an illustration of convertingalpha information to a YUV420 video frame format in an RGB color space.The stream encoder 124 may account for chroma sub-sampling by treatingalpha as YUV and adding a header of 2×3 alpha 400 to 2×2yuv2rgb(scale(alpha)) blocks 416, where the first 2×2 alpha values {a0,a1, a2, a3} map to 2×2 Y values {y0, y1, y2, y3} and the remaining 1×2alpha values {a4, a5} are replicated to 2×2 U/V values {u0, v0}, asillustrated in FIG. 4A.

For converting alpha information to a YUV444 video frame format in a YUVcolor space, the stream encoder 124 may scale alpha to according to aYUV space definition (BT.601/709, full/studio swing), then attachtriplets of successive alpha samples as a single YUV sample: a0 to y0;a1 to u0; and a2 to v0. For converting alpha information to a YUV4420video frame format in the YUV color space, the stream encoder 124 mayscale alpha according to the YUV space definition, then attach twotriplets of alpha samples as 4Y samples+2UV samples: a0, a1, a2, a3 toy0 y1, y2 y3 (4 Y samples); a4 to u0 (1 UV sample); and a5 to v0 (1 UVsample).

In transcoding the alpha information, the stream encoder 124 maymaintain full alpha resolution, and may account for different YUV spacedefinitions (e.g., BT.601/709, full/studio swing), RGB/YUV spaceoverlaps, and/or YUV420 chroma sub-sampling. Further, the total sizewith attached alpha information for a YUV444 format may be expressed as:

total size=original frame size*(1+⅓)  equation (1)

and for a YUV420 format as:

header size=original frame size*(1+⅔)  equation (2)

where the fraction part of equations (1) and (2) may refer to the costof adding an image region which carries the alpha information.

In using any color space, the stream encoder 124 may align a frame'swidth and/or height to given block sizes by replicating frameedge/border values. During decoding by the stream decoder 140, theoriginal frame sizes may still be reconstructed using “block size %frame size.” Converting the alpha information to a YUV444 video frameformat in the YUV color space or the RGB color space, as describedabove, may be fast with minimal precision loss due to (BT.601/709,full/studio swing) scaling. Further, converting alpha information to aYUV420 video frame format in an RGB color space may allow streaming ofalpha information when the underlying video pipeline does not provideYUV access on either end, or YUV color space operations are impracticalfor any reason. Thus, using disclosed approaches, the streaming ofper-pixel transparency information may be provided despite thelimitations of conventional codecs.

FIG. 4B is an illustration of an example layout 412 of image regions404A and 404B of the image 404 with per-pixel alpha information and of adisplay of the image 404 on a desktop 402, in accordance with someembodiments of the present disclosure. The image 404 is displayed on thedesktop 402 over the desktop background 408 using the per-pixel alphainformation and represents visual content of the window 406, by way ofexample. The image 404 may be displayed in a similar manner on a localdesktop or a remote desktop. Where the desktop 402 is a localdesktop—such as the local desktop 102B—the window 406 may be a proxywindow for an application window on the remote desktop.

When encoded in a video format for display on the local desktop, theimage 404 may be represented by image data in which alpha samples forpixels are encoded to the color space used to store color samples forother pixels (an RGB color space in YUV444 format in the illustratedexample). For example, the layout 412 of the image 404 is shown in FIG.4B where the image region 404A may comprise alpha samples for pixels inan image region 404B of the image 404. The image 404 may be, forexample, a splash screen of an application in which alpha samples 410Aindicate completely transparent pixels for color samples 420A and alphasamples 410B indicate complete non-transparent pixels for color samples420B. In an example where the image 404 uses an RGB color space, a colorsample for each pixel in image region 404A may store alpha samples forthree pixels in the image region 404B. Thus, the image region 404A maybe a third of the size of the image region 404B. The association betweenan alpha sample and a color sample may be implicit in the pixel locationand the used RGB channel of the alpha sample, such that the decoder mayuse the pixel location and the used RGB channel of an alpha sample tocompute the pixel location of the color sample.

Further aspects of the disclosure provide approaches that may be usedfor the transfer of visuals, such as desktop visuals, using image dataof a video stream(s) in which the visuals are split into 2D pages of avirtual frame, with each of the 2D pages being a fraction of a size of aframe of a video stream(s). The stream encoder 124 may use theseapproaches to encode the desktop visuals of the network stream 134. Indisclosed approaches, a virtual frame may be similar to a frame of thevideo stream(s) that the desktop renderer 136 may otherwise use torender one or more portions of the remote desktop 102A on the localdesktop 102B at a given time, except that regions of the virtual framemay be distributed across multiple actual frames of the video(s) streamas opposed to a single frame.

Streaming “paged” desktop visuals may simplify the logic applied to theplacement of multiple desktop elements into a single transmitted surface(e.g., the visual surface 332 of FIG. 3B) and their later reconstructionfrom that surface by the desktop renderer 136. Further, disclosedapproaches may avoid the usage of complex bin-packing algorithms. Insome examples, disclosed approaches may allow for the encoding, by thestream encoder 124, of one or more portions of a virtual frame, ratherthan complete window visuals in every frame, which may allow forbalancing workload and bandwidth requirements. Further, disclosedapproaches may be used to mitigate video frame size limitations of videocodecs used to encode the video stream of the network stream 134.

Referring now to FIG. 5, FIG. 5 is an illustration of generating a frame506 of a virtual frame from window visuals of application windows, inaccordance with some embodiments of the present disclosure. An image 504of FIG. 5 may correspond to window visuals of an application windowdesignated by the streaming determiner 126 to be streamed in the networkstream 134. For example, the image 504 may correspond to the image 404of FIG. 4B. The image 502 of FIG. 5 may correspond to window visuals ofanother application window designated by the streaming determiner 126 tobe streamed in the network stream 134. Any number of sets of desktopvisuals for any number of sets of desktop elements, such as thosecorresponding to the image 502 and the image 504 may be assigned to avirtual frame by the virtual frame generator 128. For example, the setsof desktop visuals may be designated for concurrent display on the localdesktop 102B at a given time. Thus, for the visual surface 332 of FIG.3B, the virtual frame may include window visuals for each of theapplication windows 312, 314, and 316. The virtual frame generator 128may obtain the window visuals for a virtual frame from any number ofvisual surfaces, such as the visual surface 332.

The virtual frame generator 128 may split the desktop visuals for eachdesktop element of the virtual frame into 2D pages. For example, in FIG.5, a grid is displayed over the image 504 with each cell correspondingto a 2D page, and visual content of the 2D pages forming the image 504.Similarly, a grid is displayed over the image 502 with each cellcorresponding to a 2D page, and visual content of the 2D pages formingthe image 502. This may be similar to how an operating system may splitvirtual memory ranges into one-dimensional pages of physical memory.Each of the 2D pages may be a fixed-size fraction of a video frame andmay ideally align with downstream encoder block sizes to avoid artifactsand to improve encoding performance.

A page table for mapping the 2D pages to the virtual frame may beencoded in the desktop metadata (such as window metadata) of the networkstream 134 and may be used by the desktop renderer 136 to renderappropriate portions of the virtual fame. The mapping of the page tablefor each virtual frame (e.g., a {window ID, window page} to{stream/container ID, frame/container page}) may be expressed as afunction of the other window metadata, including window page containersize, and page size. As each video stream may already provide per-framewindow metadata (as described herein), and the window page containersize may be given by the video frame size, the remote system 116 and thelocal system 118 may be easily configured to use the same page size toenable the desktop renderer 136 to reconstruct per-frame page tableswithout having to send them with each frame.

As a specific and non-limiting example, the image 502 may have aresolution of 423×323, the image 504 may have a resolution of 849×1028,the 2D page size may have a resolution of 180×180, and the frames of thevideos stream may have a resolution of 1080×1080 (e.g., the 2D page sizemay be evenly divisible into the frame size for maximum usage). In theabove example, the virtual frames may also have a resolution of1080×1080, or may have a larger resolution, such as 1920×1080, or asmaller resolution (e.g., where a frame supports multiple virtualframes).

The virtual frame generator 128 may treat each video frame as a virtualframe container (e.g., a window page container) that stores visualcontent of any number of the 2D pages for any number of the desktopelements in a 2D grid arrangement. For example, FIG. 5 depicts a frame506 in which image data of at least some of the 2D pages that correspondto the image 504 and the image 502 are stored in respective cells of the2D grid of the frame 506. Other 2D pages of the virtual frame maysimilarly be stored in respective cells of a 2D grid of other frames ofthe virtual frame. Any of the 2D grids may be resized at runtime toaccommodate page count requirements of the virtual frames, therequirements of the transport channel (e.g., and without limitation, ofthe stream encoder 124, of network bandwidth, and/or of the streamdecoder 140).

These approaches may allow for the alpha information to be encoded intothe video stream(s) while the network stream 134 still supporting highresolution desktop streaming, may allow for the network stream 134 tosupport higher resolution desktop streaming than the video codecs mayotherwise allow (e.g., the actual desktop area may exceed limits ofinvolved video codec implementations). Also, these approaches may beused to include in the network stream 134 portions of desktop visualsthat are obscured or occluded on the remote desktop 102A (e.g., byoverlapping portions of other windows). For example, while in someembodiments the relative locations of one or more of the desktopelements of the remote desktop 102A may be mirrored with the relativelocations of proxies for those elements on the local desktop 102B, inother cases they may be located independently. Thus, those portions ofthe desktop visuals may not be obscured on the local desktop 102B.

These approaches may allow for the desktop renderer 136 to display theportions desktop visuals that are obscured or occluded on the remotedesktop 102A (e.g., by overlapping portions of other windows).Additionally, or alternatively, these approaches may allow for bufferingof the portions of the desktop visuals on the local desktop 102B forwhen they become unobscured. Additionally, or alternatively, theseapproaches may allow for such features as displaying a preview window204B of FIG. 2B for the proxy window 106B in complete form on the localdesktop 102B even where portions of the desktop visuals are obscured, asshown. The preview window 204A of FIG. 2A and the preview window 204B ofFIG. 2B, which may also be referred to as thumbnail windows, display asmaller representation of the application window 106A and/or the proxywindow 106B. The preview window 204B may, for example, be independentfrom the preview window 204A of FIG. 2A allowing for preview windowseven where the remote desktop 102A does not support preview windows orimage data of a preview window is otherwise unavailable. As anotherexample, the preview window 204B may be a proxy window for the previewwindow 204A.

In various embodiments, the network stream 134 may include multipleparallel video streams to send each virtual frame over the network bydistributing the fixed-size fractions (e.g., pages) of each virtualframe over multiple page containers. This approach may allow the systemto mitigate video frame size or other limitations of the video codecsused to perform the encoding, where video streams may be saturatedsequentially, or in parallel (e.g., to balance workload and bandwidthrequirements).

The desktop event monitor 132A of the remote system 116 may beconfigured to identify events associated with the remote desktop 102A,such as a window creation event or a window destruction event associatedwith an application and/or application window on the remote desktop102A. For example, an application may launch the application window106A. The desktop event monitor 132A may notify the streaming determiner126, which may include the application window 106A in the network stream134 based at least in part on a detected window creation event.Conversely, an application window associated with an application may bedestroyed (e.g., closed by the user, system, or application). Thedesktop event monitor 132A may notify the streaming determiner 126,which may exclude the application window 106A from the network stream134 based at least in part on a detected window destruction event.

As another example, the desktop event monitor 132A may be configured toidentify a window mode change event of an application and/or applicationwindow on the remote desktop 102A. For example, an application mayswitch the application window 106A to a full screen mode, in which casethe streaming determiner 126 may exclude one or more other desktopelements from the network stream 134 based at least in part on thedetected window mode change event. As another example, the applicationmay switch the application window 106A to a windowed screen mode, inwhich case the streaming determiner 126 may add one or more otherdesktop elements to the network stream 134 based at least in part on thedetected window mode change event. The desktop event monitor 132A may beimplemented, at least partially, using a frame provider that usesexisting screen monitor logic to track the screen-state (e.g., fullscreen mode, windowed mode) of a running application, where the logicmay automatically switch between windowed and full-screen capture asneeded. In some examples, the desktop event monitor 132A may use awindow lifecycle event monitor provided by the operating system, forexample and without limitation. In other examples, the desktop eventmonitor 132A may be implemented by polling current window lifecyclestates. The streaming determiner 126 may maintain a list of processes ofinterest. On every window create/destruction event, the window lifecycleevent monitor (or other event tracking module of the desktop eventmonitor 132A) may be used to update the list of active windows and thestreaming determiner 126 may generate a list of windows to include orexclude from the network stream 134.

In some examples, the desktop event monitor 132A of the remote system116 may be configured to identify or detect a window creation event, awindow destruction event, a window mode change events, and/or otherevent associated with an application and/or application window on theremote desktop 102A that is initiated by user input to the local desktop102B, by an application, and/or operating system running on the localsystem 118. The desktop event monitor 132A may also be configured toidentify or detect mouse or cursor events (e.g., mouse clicks, cursormotion, etc.) initiated by user input to the local desktop 102B. To doso, the desktop event monitor 132A may work in conjunction with thedesktop event monitor 132B of the local system 118. For example, thedesktop event monitor 132B may detect or identify the events, which maybe transmitted to the desktop event monitor 132A in the network stream134 (e.g., as window properties which may employ delta-encoding). Otherexamples of events that may be detected by the desktop event monitor132B include gain/lose focus events, move/resize commands, andminimize/maximize/restore commands.

The desktop event monitor 132B may identify the transmitted event(s) andthe desktop event monitor 132A may implement the event(s) at the remotedesktop 102A (e.g., by simulating user input to the remote desktop 102Aand sending one or more commands to the operating system or application,etc.). As an example, user input to the command button 220 of the proxywindow 106B of FIG. 2B may result in the desktop event monitor 132Bdetecting a window destruction event for the application window 106A ofFIG. 2A. The desktop event monitor 132B may provide the event to thecommunications manager 120B, which may transmit the event to the remotesystem 116. The desktop event monitor 132A may detect the transmittedevent and cause the application window to be closed on the remote system116. As another example, the desktop event monitor 132A may use mouseclick events from the desktop event monitor 132B to generatecorresponding events at the remote system 116, such as those associatedwith applications and/or application windows (e.g., to launch or closeapplication windows, etc.).

In any example, the streaming determiner 126 may update a list ofapplication windows and/or other desktop elements to include in thenetwork stream 134 to be transmitted to the local desktop 102B inresponse to being notified by the desktop even monitor 132A of theidentified events.

At the local system 118, the communications manager 120B, may receivethe network stream 134. The network stream 134 may include datarepresentative of desktop visuals and corresponding desktop propertiesfor any number of desktop elements (e.g., in the form of desktopmetadata). The data representative of desktop visuals and correspondingdesktop properties may then be decoded by the stream decoder 140. Thedata representative of desktop visuals may include alpha information,such as at least some of the alpha samples 510A of FIG. 5 (e.g., fromthe frame 506 or from the visual surface 332 of FIG. 3B or other visualsurface). Where alpha samples are included in the network stream 134,the desktop renderer 136 may correlate the alpha samples with the colorsamples in order to render pixels with appropriate transparency.

The desktop renderer 136 may render, on the local desktop 102B, proxydesktop elements of the desktop elements of the remote desktop 102A tothe local desktop 102B using at least some of the decoded desktopvisuals and their corresponding decoded desktop properties. For example,at least some of the window visuals for the application window 106A maybe rendered in the proxy window 106B, as in FIG. 2B, based at least inpart on the window properties (e.g., the location, size, etc.). In someexamples, the application window 106A may be on the remote desktop 102Aat one coordinate location but is rendered on the local desktop 102B ata different coordinate location. For example, the proxy window 106B maybe located differently from the application window 106A. Additionally,or alternatively, the local desktop 102B may use a different coordinatesystem than then remote desktop 102A. In these cases, the location fromthe window properties may be translated by the desktop renderer 136 tothe coordinate system of the local desktop 102B, or the translation maybe performed by the desktop capturer 130 prior to inclusion in thenetwork stream 134. In some examples, the desktop renderer 136 may applycoordinates of desktop elements that are associated with video frames(e.g., in the desktop metadata) to decoded video frames, in order tocrop all desktop elements from the frames and then reconstruct theremote desktop 102A layout or topology including the application windowlayout to replicate the layout from the remote system 116.

As a further example of rendering proxy desktop elements, at least someof the window visuals for the application window 216A of FIG. 2A may berendered in a proxy window 216B of FIG. 2B. When rendering the proxydesktop elements, the desktop renderer 136 may preserve the relativeZ-order of the corresponding application windows on the remote desktop102A. For example, the proxy window 216B may be displayed over the proxywindow 106B on the local desktop 102B in FIG. 2B because the applicationwindow 216A is displayed over the application window 106A on the remotedesktop 102A in FIG. 2A. The relative Z-order of application windowsand/or other desktop elements that is used by the desktop renderer 136may be captured by window properties that may be included in the windowmetadata, for example.

Rendered proxy desktop elements may be present on the local desktop 102Balong with native desktop elements and may be seamlessly integrated intothe local desktop 102B, such that they may behave like native desktopelements. For example, as in FIG. 2B, the local desktop 102B may includethe proxy window 106B as well as a native application window 222 whichis native to the local device. Moreover, the proxy window 106B and thenative application window 114 may include one or more different systemareas or frame areas when the local desktop 102B is on a differentoperating system than the remote desktop 102A (e.g., a Windows formattedproxy window may be present on a Linux operating system with other Linuxformatted application windows). As another example, the desktop renderer136 may render one or more system areas or frame areas of one or moreproxy desktop elements using native desktop visuals. A user may interactwith both the proxy window 106B and the native application window 114 onthe local desktop 102B as though both are native to the local desktop102B. For example, the desktop renderer 136 may track the Z-order ofproxy windows and/or other proxy desktop elements relative to nativeapplication windows and/or desktop elements and display the desktopvisuals accordingly (e.g., accounting for overlaps, changes in Z-order,etc.). Other native elements may be provided such as a taskbar 210B, orthe taskbar 210B may be a proxy desktop element for the taskbar 210A.

The local desktop 102B may include a cursor 208B that is native to thelocal system 118 or may mimic the cursor 208A of the local system 118.Many applications define their own specific mouse cursors. For example,while a uni-directional arrow is a standard cursor during normaloperation for standard productivity applications such as web-browsingand word processing software, the cursor 208A may be represented bydifferent graphical shapes in different applications or under differentoperating circumstances. As an example, when zoom functionality isactive, the cursor 208A may have the shape of a magnifying glass.Similarly, the cursor 208A may have the shape of an hourglass duringperiods of longer than normal loading or heavy processing. Gamingapplications may also use different shapes to represent the cursor 208A(e.g., an icon of a sword or a reticule). Conventional remote desktopstreaming approaches may not be able to mimic the appropriate cursorshape effectively.

In some embodiments, the cursor 208A may be used as a desktop elementallowing for display of even non-standard graphical shapes as cursors onthe local desktop 102B. The desktop capturer 130 may capture the handle(or other identifier) of the current cursor 208A on the remote system116 (e.g., as defined by the corresponding application). The handle maythen be referenced in a dataset of known cursors at the remote system116 to determine a corresponding identifier that is sent to the localsystem 118. The desktop renderer 136 of the local system 118 mayreference the identifier in its own dataset of known cursors and set thelocal cursor 208B to the appropriate known cursor. When the applicationcursor is not found in the set of known cursors at the remote system116, the new cursor may be captured by the desktop capturer 130 (e.g.,as a bitmap), cached both at the remote system 116 and the local system118, and then appended to the set of known cursors for futurereference/usage. The remote system 116 may intelligently transmit onlycaptured bitmaps of new cursors; as the old/known ones may already besaved into a cache at the local system 118.

Referring now to FIG. 6, FIG. 6 is a flow diagram showing a method 600for remote desktop streaming, in accordance with some embodiments of thepresent disclosure. At block B602, the method 600 includes determiningat least one application window of a remote desktop to stream. Forexample, the streaming determiner 126 of the remote system 116 maydetermine at least one application window of the remote desktop 102A toinclude in the network stream 134 to the local desktop 102B of a localdevice, such as the application window 106A.

At block B604, the method 600 includes capturing window visuals andwindow properties of the at least one application window. For example,the desktop capturer 130 of the remote system 116 may capture windowvisuals and corresponding window properties of the at least oneapplication window.

At block B606, the method 600 includes encoding the window visuals andwindow properties to a stream. For example, the stream encoder 124 ofthe remote system 116 may encode the window visuals and the windowproperties of at least one application window to at least one videostream of the network stream 134.

At block B608, the method 600 includes transmitting at least some of thestream to a local desktop. For example, the communications manager 120Aof the remote system 116 may transmit the network stream 134 comprisingthe at least one video stream and window metadata representative of thewindow properties of the at least one application window to the localdesktop 102B.

Referring now to FIG. 7, FIG. 7 is a flow diagram showing a method 700for rendering desktop visuals of a remote desktop on a local desktop, inaccordance with some embodiments of the present disclosure.

At block B702, the method 700 includes receiving an encoded stream. Forexample, the communications manager 120B of the local system 118 mayreceive the network stream 134 of one or more application windows of theremote desktop 102A to the local desktop 102B of a local device, such asthe application window 106A. The network stream 134 may include windowmetadata representative of window properties of the one or moreapplication windows and window visuals of the one or more applicationwindows encoded in at least one video stream of the network stream 134.

At block B704, the method 700 includes decoding window visuals andwindow metadata associated with at least one application window from thestream. For example, the stream decoder 140 of the local system 118 maydecode the window visuals and the window metadata from the networkstream 134.

At block B706, the method 700 includes rendering the window visuals on alocal desktop using the window metadata to construct proxy window(s).For example, the desktop renderer 136 of the local system 118 mayrender—on the local desktop 102B—the decoded window visuals thatcorrespond to an application window of the one or more applicationwindows in a proxy window of the application window (e.g., the proxywindow 106B of the application window 106A) based at least in part onthe decoded window properties of the application window.

Referring now to FIG. 8, FIG. 8 is a flow diagram showing a method 800for selectively transmitting portions of a remote desktop to a localdesktop, in accordance with some embodiments of the present disclosure.At block B802, the method 800 includes determining at least oneapplication window of a remote desktop to include in a stream. Forexample, the streaming determiner 126 of the remote system 116 maydetermine at least one application window of the remote desktop 102A toinclude in the network stream 134 to the local desktop 102B of a localdevice, such as the application window 106A.

At block B804, the method 800 includes determining a portion of theremote desktop to exclude from the stream. For example, the desktopcapturer 130 may use the streaming determiner 126 of the remote system116 to determine at least the desktop region 310 (FIG. 3A) of the remotedesktop 102A to exclude from the network stream 134.

At block B806, the method 800 includes removing the portion of theremote desktop from a first (e.g., a captured) visual surface togenerate a second (e.g., to be transmitted) visual surface. For example,the desktop capturer 130 may remove the desktop region 310 of the remotedesktop 102A from the remote desktop surface 304 to generate the visualsurface 334 having the region 330 in place of the desktop region 310.This may or may not involve a stencil and may or may not involvereducing the entropy of the desktop region 310 to generate the region330. Further, while the remote desktop surface 304 is used as anexample, the portion(s) of the remote desktop 102A may be removed fromother visual surfaces that may be used to encode desktop visuals to thenetwork stream 134.

At block B808, the method 800 includes encoding the second visualsurface to the stream. For example, the stream encoder 124 of the remotesystem 116 may encode, to the network stream 134, the visual surface 334that comprises window visuals of at least one application window withoutthe desktop region 310 of the remote desktop 102A.

At block B810, the method 800 includes transmitting the stream to alocal desktop. For example, the communications manager 120A of theremote system 116 may transmit the network stream 134 to the localdesktop 102B.

FIG. 9 is a flow diagram showing a method 900 for transmitting alphainformation of visuals in one or more frames of a video stream, inaccordance with some embodiments of the present disclosure. The method900, at block B902 includes determining visuals to include in a stream.For example, the streaming determiner 126 of the remote system 116 maydetermine to include window visuals of the application windowcorresponding to the image 504 of FIG. 5 in the network stream 134.

The method 900, at block B904 includes transcoding alpha information ofthe visuals to the color space of the video stream, to generate the“alpha” samples. For example, the stream encoder 124 of the remotesystem 116 may transcode alpha information of the visuals to an RGB orYUV color space to generate the alpha samples 510A of FIG. 5.

The method 900, at block B906 includes storing the alpha samples inimage data of at least one video frame of the stream. For example, thestream encoder 124 of the remote system 116 may store the alpha samplesin image data of the frame 506.

The method 900, at block B908 includes encoding the at least one videoframe to the stream. This video frame may include transcoded alphainformation. For example, the stream encoder 124 of the remote system116 may encode the frame 506 to the network stream 134.

The method 900, at block B910 includes transmitting the stream to a userdevice. For example, the communications manager 120A of the remotesystem 116 may transmit the network stream 134 to a local device thathosts the local desktop 102B. While examples described are applicable toremote desktop streaming, the method 900 may more generally be used tostream and display visuals for any suitable applications.

FIG. 10 is a flow diagram showing a method 1000 for streaming a virtualframe over a video stream, in accordance with some embodiments of thepresent disclosure. The method 1000, at block B1002 includes determiningvisuals to include in a stream. For example, the streaming determiner126 of the remote system 116 may determine to include window visuals ofthe application windows corresponding to the image 502 and the image 504of FIG. 5 in the network stream 134.

The method 1000, at block B1004 includes splitting the visuals intopages of a virtual frame with each page being a fraction of a size ofeach of at least one video frame of one or more video streams of thestream. For example, the stream encoder 124 of the remote system 116 maysplit the visuals corresponding to the image 502 and the image 504 intopages of a virtual frame, as indicated in FIG. 5. Each page may be afraction of a size of the frame 506 of a video stream of the networkstream 134. Each page may be sized optimally for the currently activevideo encoder.

The method 1000, at block B1006 includes storing at least some of thepages in a video frame of the stream. For example, the stream encoder124 of the remote system 116 may store the pages in the frame 506 of thenetwork stream 134. In various examples, a subset of the pages may bestored in one video frame and others of the pages may be stored in othervideo frames of the network stream 134. Further, the stream encoder 124may store the pages in the video frame in accordance with the pagetable. The network stream 134 may include metadata (e.g.,stream/container ID, frame/container page ID, etc.) that the localsystem 118 may use to determine where each page of the received virtualframe maps into proxy windows, in accordance with the page table.

The method 1000, at block B1008 includes encoding the video frame to thestream. For example, the stream encoder 124 of the remote system 116 mayencode the frame 506 to the network stream 134.

The method 1000, at block B1010 includes transmitting the stream to auser device. For example, the communications manager 120A of the remotesystem 116 may transmit the network stream 134 to a local device thathosts the local desktop 102B. While examples described are applicable toremote desktop streaming, the method 1000 may more generally be used tostream and display visuals for any suitable applications.

The desktop streaming system 100 may be implemented in an exampleoperating environment 1100 of FIG. 11, in accordance with someembodiments of the present disclosure. Among other components notillustrated, the operating environment 1100 may include a serverdevice(s) 1102, a network(s) 1104, a user device(s) 1106, and a datastore(s) 1108. It should be understood that operating environment 1100shown in FIG. 11 is an example of one suitable operating environment.Each of the components shown in FIG. 11 may be implemented via any typeof computing device, such as one or more of computing device 1200described in connection with FIG. 12, for example. These components maycommunicate with each other via the network 1104, which may be of anytype. The network 1104 may include multiple networks, or a network ofnetworks, but is illustrated in simple form so as not to obscure aspectsof the present disclosure. The network 1104 may include any number ofdifferent network topologies.

It should be understood that any number of the user device(s) 1106, theserver devices 1102, and the data stores 1108 may be employed within theoperating environment 1100 within the scope of the present disclosure.Each may be configured as a single device or multiple devicescooperating in a distributed environment.

The user device(s) 1106 may include at least some of the components,features, and functionality of the example computing device 1200described herein with respect to FIG. 12. By way of example and notlimitation, a user device 1106 may be embodied as a personal computer(PC), a laptop computer, a mobile device, a smartphone, a tabletcomputer, a smart watch, a wearable computer, a personal digitalassistant (PDA), an MP3 player, a global positioning system (GPS) ordevice, a video player, a handheld communications device, a gamingdevice or system, an entertainment system, a vehicle computer system, anembedded system controller, a remote control, an appliance, a consumerelectronic device, a workstation, any combination of these delineateddevices, or any other suitable device.

The user device(s) 1106 may include one or more processors, and one ormore computer-readable media. The computer-readable media may includecomputer-readable instructions executable by the one or more processors.The instructions may, when executed by the one or more processors, causethe one or more processors to perform any combination and/or portion ofthe methods described herein and/or implement any portion of thefunctionality of the desktop streaming system 100 of FIG. 1.

The server device(s) 1102 may also include one or more processors, andone or more computer-readable media. The computer-readable mediaincludes computer-readable instructions executable by the one or moreprocessors. The instructions may, when executed by the one or moreprocessors, cause the one or more processors to perform any combinationand/or portion of the methods described herein and/or implement anyportion of the functionality of the desktop streaming system 100 of FIG.1.

The data store(s) 1108 may comprise one or more computer-readable media.The computer-readable media may include computer-readable instructionsexecutable by the one or more processors. The instructions may, whenexecuted by the one or more processors, cause the one or more processorsto perform any combination and/or portion of the methods describedherein and/or implement any portion of the functionality of the desktopstreaming system 100 of FIG. 1. The data store(s) 1108 (or computer datastorage) is depicted as a single component, but may be embodied as oneor more data stores (e.g., databases) and may be at least partially inthe cloud.

Although depicted external to the server device(s) 1102 and the userdevice(s) 1106, the data store(s) 1108 may be at least partiallyembodied on any combination of the server device(s) 1102 and/or the userdevice(s) 1106 (e.g., as memory 1204 (FIG. 12)). For example, someinformation may be stored on a user device(s) 1106, and other and/orduplicate information may be stored externally (e.g., on a serverdevice(s) 1102). Thus, it should be appreciated that information in thedata store(s) 1108 may be distributed in any suitable manner across oneor more data stores for storage (which may be hosted externally). Forexample, the data store(s) 1108 may comprise at least some of the one ormore computer-readable media of the server device(s) 1102 and/or atleast some of the one or more computer-readable media of the userdevice(s) 1106.

Referring now to FIG. 12, FIG. 12 is a block diagram of an examplecomputing device 1200 suitable for use in implementing some embodimentsof the present disclosure. Computing device 1200 may include a bus 1202that directly or indirectly couples the following devices: memory 1204,one or more central processing units (CPUs) 1206, one or more graphicsprocessing units (GPUs) 1208, a communication interface 1210,input/output (I/O) ports 1212, input/output components 1214, a powersupply 1216, and one or more presentation components 1218 (e.g.,display(s)).

Although the various blocks of FIG. 12 are shown as connected via thebus 1202 with lines, this is not intended to be limiting and is forclarity only. For example, in some embodiments, a presentation component1218, such as a display device, may be considered an I/O component 1214(e.g., if the display is a touch screen). As another example, the CPUs1206 and/or GPUs 1208 may include memory (e.g., the memory 1204 may berepresentative of a storage device in addition to the memory of the GPUs1208, the CPUs 1206, and/or other components). In other words, thecomputing device of FIG. 12 is merely illustrative. Distinction is notmade between such categories as “workstation,” “server,” “laptop,”“desktop,” “tablet,” “client device,” “mobile device,” “hand-helddevice,” “game console,” “electronic control unit (ECU),” “virtualreality system,” and/or other device or system types, as all arecontemplated within the scope of the computing device of FIG. 12.

The bus 1202 may represent one or more busses, such as an address bus, adata bus, a control bus, or a combination thereof. The bus 1202 mayinclude one or more bus types, such as an industry standard architecture(ISA) bus, an extended industry standard architecture (EISA) bus, avideo electronics standards association (VESA) bus, a peripheralcomponent interconnect (PCI) bus, a peripheral component interconnectexpress (PCIe) bus, and/or another type of bus.

The memory 1204 may include any of a variety of computer-readable media.The computer-readable media may be any available media that may beaccessed by the computing device 1200. The computer-readable media mayinclude both volatile and nonvolatile media, and removable andnon-removable media. By way of example, and not limitation, thecomputer-readable media may comprise computer-storage media andcommunication media.

The computer-storage media may include both volatile and nonvolatilemedia and/or removable and non-removable media implemented in any methodor technology for storage of information such as computer-readableinstructions, data structures, program modules, and/or other data types.For example, the memory 1204 may store computer-readable instructions(e.g., that represent a program(s) and/or a program element(s), such asan operating system. Computer-storage media may include, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which may be used to storethe desired information and which may be accessed by computing device1200. As used herein, computer storage media does not comprise signalsper se.

The communication media may embody computer-readable instructions, datastructures, program modules, and/or other data types in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” mayrefer to a signal that has one or more of its characteristics set orchanged in such a manner as to encode information in the signal. By wayof example, and not limitation, the communication media may includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, RF, infrared and other wireless media.Combinations of any of the above should also be included within thescope of computer-readable media.

The CPU(s) 1206 may be configured to execute the computer-readableinstructions to control one or more components of the computing device1200 to perform one or more of the methods and/or processes describedherein. The CPU(s) 1206 may each include one or more cores (e.g., one,two, four, eight, twenty-eight, seventy-two, etc.) that are capable ofhandling a multitude of software threads simultaneously. The CPU(s) 1206may include any type of processor, and may include different types ofprocessors depending on the type of computing device 1200 implemented(e.g., processors with fewer cores for mobile devices and processorswith more cores for servers). For example, depending on the type ofcomputing device 1200, the processor may be an ARM processor implementedusing Reduced Instruction Set Computing (RISC) or an x86 processorimplemented using Complex Instruction Set Computing (CISC). Thecomputing device 1200 may include one or more CPUs 1206 in addition toone or more microprocessors or supplementary co-processors, such as mathco-processors.

The GPU(s) 1208 may be used by the computing device 1200 to rendergraphics (e.g., 3D graphics). The GPU(s) 1208 may include hundreds orthousands of cores that are capable of handling hundreds or thousands ofsoftware threads simultaneously. The GPU(s) 1208 may generate pixel datafor output images in response to rendering commands (e.g., renderingcommands from the CPU(s) 1206 received via a host interface). The GPU(s)1208 may include graphics memory, such as display memory, for storingpixel data. The display memory may be included as part of the memory1204. The GPU(s) 1208 may include two or more GPUs operating in parallel(e.g., via a link). When combined together, each GPU 1208 may generatepixel data for different portions of an output image or for differentoutput images (e.g., a first GPU for a first image and a second GPU fora second image). Each GPU may include its own memory, or may sharememory with other GPUs.

In examples where the computing device 1200 does not include the GPU(s)1208, the CPU(s) 1206 may be used to render graphics.

The communication interface 1210 may include one or more receivers,transmitters, and/or transceivers that enable the computing device 1200to communicate with other computing devices via an electroniccommunication network, included wired and/or wireless communications.The communication interface 1210 may include components andfunctionality to enable communication over any of a number of differentnetworks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth,Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating overEthernet), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.),and/or the Internet.

The I/O ports 1212 may enable the computing device 1200 to be logicallycoupled to other devices including the I/O components 1214, thepresentation component(s) 1218, and/or other components, some of whichmay be built in to (e.g., integrated in) the computing device 1200.Illustrative I/O components 1214 include a microphone, mouse, keyboard,joystick, game pad, game controller, satellite dish, scanner, printer,wireless device, etc. The I/O components 1214 may provide a natural userinterface (NUI) that processes air gestures, voice, or otherphysiological inputs generated by a user. In some instances, inputs maybe transmitted to an appropriate network element for further processing.An NUI may implement any combination of speech recognition, stylusrecognition, facial recognition, biometric recognition, gesturerecognition both on screen and adjacent to the screen, air gestures,head and eye tracking, and touch recognition (as described in moredetail below) associated with a display of the computing device 1200.The computing device 1200 may be include depth cameras, such asstereoscopic camera systems, infrared camera systems, RGB camerasystems, touchscreen technology, and combinations of these, for gesturedetection and recognition. Additionally, the computing device 1200 mayinclude accelerometers or gyroscopes (e.g., as part of an inertiameasurement unit (IMU)) that enable detection of motion. In someexamples, the output of the accelerometers or gyroscopes may be used bythe computing device 1200 to render immersive augmented reality orvirtual reality.

The power supply 1216 may include a hard-wired power supply, a batterypower supply, or a combination thereof. The power supply 1216 mayprovide power to the computing device 1200 to enable the components ofthe computing device 1200 to operate.

The presentation component(s) 1218 may include a display (e.g., amonitor, a touch screen, a television screen, a heads-up-display (HUD),other display types, or a combination thereof), speakers, and/or otherpresentation components. The presentation component(s) 1218 may receivedata from other components (e.g., the GPU(s) 1208, the CPU(s) 1206,etc.), and output the data (e.g., as an image, video, sound, etc.).

The disclosure may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Thedisclosure may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The disclosure mayalso be practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

As used herein, a recitation of “and/or” with respect to two or moreelements should be interpreted to mean only one element, or acombination of elements. For example, “element A, element B, and/orelement C” may include only element A, only element B, only element C,element A and element B, element A and element C, element B and elementC, or elements A, B, and C. In addition, “at least one of element A orelement B” may include at least one of element A, at least one ofelement B, or at least one of element A and at least one of element B.Further, “at least one of element A and element B” may include at leastone of element A, at least one of element B, or at least one of elementA and at least one of element B.

The subject matter of the present disclosure is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of thisdisclosure. Rather, the inventors have contemplated that the claimedsubject matter might also be embodied in other ways, to includedifferent steps or combinations of steps similar to the ones describedin this document, in conjunction with other present or futuretechnologies. Moreover, although the terms “step” and/or “block” may beused herein to connote different elements of methods employed, the termsshould not be interpreted as implying any particular order among orbetween various steps herein disclosed unless and except when the orderof individual steps is explicitly described.

What is claimed is:
 1. A method comprising: determining applicationvisuals to include in a video stream; allocating the application visualsamong pages of a virtual frame; storing at least one page of the pagesin image data representative of one or more video frames of the videostream; and generating at least a portion of the video stream, the videostream comprising the image data and a page table defining mappingsbetween the at least one page and a respective position of the at leastone page in the virtual frame.
 2. The method of claim 1, wherein thevirtual frame has greater resolution than each of the one or more videoframes.
 3. The method of claim 1, wherein at least some pages of thepages are arranged in a grid of cells in the one or more video frames,each page corresponding to a particular cell of the grid of cells. 4.The method of claim 1, wherein each of the pages is a fixed-sizefraction of the one or more video frames.
 5. The method of claim 1,wherein the page table is in at least one header of the video stream. 6.The method of claim 1, wherein the storing comprises storing at least afirst page of the pages in a first portion of the image datarepresentative of a first video frame of the one or more video framesand storing at least a second page of the pages in a second portion ofthe image data representative of a second video frame of the one or morevideo frames.
 7. The method of claim 1, wherein the storing comprisesstoring at least a first page of the pages in the image datarepresentative of the one or more video frames of the video stream, andthe method further includes: storing at least a second page of the pagesin second image data representative of one or more second video framesof a second video stream; and generating at least a portion of thesecond video stream comprising the second image data.
 8. The method ofclaim 1, wherein the page table includes a visual element identifierthat maps the at least one page to a visual element of the applicationvisuals.
 9. The method of claim 1, wherein the at least one pageincludes alpha samples of the application visuals stored in a colorspace of the video stream.
 10. The method of claim 1, wherein the atleast one page includes a plurality of the pages distributed across aplurality of the one or more video frames.
 11. A method comprising:receiving, in a video stream, image data representative of one or morevideo frames of a video stream, the one or more video frames comprisingapplication visuals stored at least partially as one or more pages of avirtual frame; reconstructing the virtual frame from the one or morepages based at least in part on a page table comprising mappings betweenthe one or more pages and respective one or more positions of the one ormore pages in the virtual frame; and rendering the application visualsof the virtual frame using the one or more pages and the mappings. 12.The method of claim 11, wherein the one or more pages are arranged in agrid of cells of the virtual frame in the one or more video frames. 13.The method of claim 11, wherein each page of the virtual frame is afixed-size fraction of the one or more video frames.
 14. The method ofclaim 11, wherein the rendering of the application visuals includesrendering at least a portion of a visual element of the applicationvisuals based at least in part on a visual element identifier extractedfrom metadata of the video stream that maps the one or more pages to thevisual element.
 15. The method of claim 11, wherein the reconstructinguses a first page of the virtual frame extracted from the image data anda second page of the virtual frame extracted from second image datarepresentative of second one or more video frames of a second videostream.
 16. A system comprising: a computing device including one ormore processing devices and one or more memory devices communicativelycoupled to the one or more processing devices storing programmedinstructions thereon, which when executed by the one or more processingdevices causes the instantiation of: a communications manager toreceive, in a video stream, image data representative of one or morevideo frames of a video stream, the one or more video frames comprisingone or more pages of a virtual frame that are representative of at leasta portion of a visual element; a renderer to: reconstruct the virtualelement from the one or more pages based at least in part on mappingsbetween the one or more pages and respective one or more positions ofthe one or more pages in the virtual frame; and render the visualelement of the virtual frame using the one or more pages and themappings.
 17. The system of claim 16, wherein the one or more pages arearranged in a grid in the one or more video frames.
 18. The system ofclaim 16, wherein the renderer renders the visual element based at leastin part on a visual element identifier extracted from metadata of thevideo stream that maps the one or more pages to the visual element. 19.The system of claim 16, wherein the renderer renders the visual elementof the virtual frame over at least one image generated by the computingdevice.
 20. The system of claim 16, wherein the reconstructing uses afirst page extracted from the image data and a second page extractedfrom second image data representative of second one or more video framesof a second video stream.